Repository navigation
Fix discovery bootstrap cache access and failure diagnostics - #409
Draft
almaslennikov wants to merge 1 commit into
Draft
almaslennikov wants to merge 1 commit into
almaslennikov wants to merge 1 commit into
Conversation
Grant the bootstrap daemon the ConfigMap read verbs required by its cache. Report unverified NIC probes and pending-publisher failure evidence without changing trust classification or device omission policy. Related to #408. Signed-off-by: Alexander Maslennikov <amaslennikov@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bootstrap daemon can become Ready before NIC initialization finishes, then crash while Launch Kit shows only the device-count wait. Its cluster-wide ConfigMap cache also lacks
list/watchpermissions in Launch Kit's bootstrap role.This change grants ConfigMap
get/list/watchwithout write verbs, reports unverified eligibility-probe exclusions by node/pod, and refreshes bootstrap pod failure evidence for missing publishers during the NicDevice wait. Diagnostics are deduplicated, deadline-bound and nonfatal so a restarting publisher can recover. Timeout errors name the last missing publishers, and progress counts only the expected node set. Existing-daemon/read-only waiters acquire no pod-status operations.Related to #408. This is a scoped correction, not a complete fix for readable ConnectX inventory being lost when BlueField startup discovery aborts. The existing trust probe and
SKIP_DEVICE_ON_DISCOVERY_ERRORpolicy remain unchanged.Evidence and compatibility
6c54e0f870f0d4ccc91fe4a5ec84440d86e587dd, merged fix: exclude restricted BlueFields from discovery #246/fix(discovery): report host-restricted BlueField exclusions #341, and all issue comments.26.7resolves to the reportednetwork-operator-v26.7.1daemon tag. Its tagged source registers a cluster-wide ConfigMap cache; bootstrap's direct-Get permission cannot satisfy its informer.Updated the existing discovery and troubleshooting guides, their skills/references, and the discovery-inventory contract. Regression cases cover exact read-only RBAC, live crash/recovery, repeated failure suppression, replacement pod UID, namespace/owner/pending-node scope, successful init completion, cancellation/deadlines, primary-error preservation, missing-node errors and unchanged read-only polling.
Validation
Local macOS/arm64, Go 1.26.2:
make build: passed.make test: passed. The initial sandbox denied local httptest listeners; rerunning with loopback access passed.make lint, pinned golangci-lint v2.14.0: zero issues.go test -race -count=1 ./pkg/networkoperatorplugin/discovery ./pkg/nicconfigdaemon: passed on the final code.python3 scripts/check-docs.py --binary build/l8k: passed, including 247 CLI/help examples, navigation and offline rendering.mkdocs build --strict: passed using the pinned documentation requirements in an isolated virtual environment.git diff --check: passed.go test -race -timeout=45m -count=1 ./...: passed. The final affected-package race run above also covers the completed-init and cancellation regression cases added during final review.Two independent final code/design reviews approved this scoped correction. Hosted CI build (including
make build-all), lint, documentation and CodeQL checks have passed. The hosted Linux/Go 1.27 race-and-coverage test job is still running; this is separate from the completed local race checks.Draft review questions and remaining evidence
The narrower RBAC-only alternative would leave the reported opaque wait and node-count mismatch undiagnosed. The observer keeps those diagnostics local to bootstrap without changing shared read-only discovery behavior.
No live cluster, GB300/arm64 or BlueField validation was performed. Image digest-to-source provenance, actual cache authorization/synchronization, affected hardware behavior and the exact dropped-node cause remain unverified. Hosted CI and cross-platform qualification are separate from local checks. Keep this PR draft until reviewers agree on the read-permission scope and diagnostic approach and the relevant runtime evidence is assessed.